Spark Configurations to Optimize Decision Tree Classification on UNSW-NB15

This paper looks at the impact of changing Spark’s configuration parameters on machine learning algorithms using a large dataset—the UNSW-NB15 dataset. The environmental conditions that will optimize classification process are studied. To build smart intrusion detection systems, deep understanding is necessary. Specifically, focus following parameters: executor memory, number executors, cores per executor, execution time, as well statistical measures. Hence, objective was to resource usage and minimize processing time for Decision Tree classification, Spark. shows whether additional resources increase performance, lower computing resources. dataset, being provides enough data complexity see changes in configurations Principal Component Analysis used preprocessing Results indicated lack executors result wasted long time. Excessive allocation did not improve Environmental tuning has noticeable impact.